* data: Intermediate layer for versioning of datasets (PyPSA#1675)

* code: Versioning for hotmaps dataset

* code: Fix CLI not working for foreign owned records

* code: versioning for co2stop dataset

* code: versioning for nitrogen statistics datset

* code: fix wrong file name for REUSE.toml

* code: versioning for NUTS 2013 and NUTS 2021

* code: Add limited debug mode for CLI script

* code: Fix issues with datasets not being detected

* code: Fix minor bugs on CLI and outputs

* code: Versioning for eurostat energy balances

* code: Remove obsolete retrieve script, now part of retrieve.smk

* code: Versioning for eurostat household energy balances

* code: Fix missing replacements for eurostat energy balances

* code: Remove prompt for API key

* code: Fix typo for enspreso biomass dataset

* code: Fix issue with OSM dataset versioning

* code: Versioning for WDPA and WDPA marine

* code: Implement versioning for TYNDP base network

* code: Fix bug missing osm_version in base_network.py

* code: Properly fix the bug this time

* code: Implement versioning for powerplants / powerplantsmatching

* code/docs: Implement versioning for costs

* code: Implement versioning for cutouts, move to data/ , deprecate shared_cutouts

* Fix: Wrong URL for powerplants

* code: Implement versioning for HDD and runoff per country (retrieval only)

* code: Add build options for HDD and runoff per country (untested)

* code: Add option for monthly retrieval of cutouts while building

* code: Restore build_natura_raster.py script

* code: Implement versioning for natura raster + reintegrate build natura raster script

* data: Change URB POP dataset to Zenodo link

* code: Adjust link for GEM GSPT

* code: Versioning for GEM Europe Gas Tracker

* doc: Correct license attributions in REUSE.toml for data/

* code: Implement versioning for BFS vehicle data

* code: Implement versioning for BFS GDP and POP data

* doc: Update data_inventory.csv for all the datasets added

* code: Fix versions for HDD and runoff per country

* code: Fix rules for creating HDD and runoff per country

* code: Add special cutout config for HDD/runoff

* code: Fix monthly_requests for cutouts

* code: Add logic for building and versioning of mobility profiles from BASt

* data: Update data for mobility profiles from BASt

* code: Windows compatability for cutouts

* code: Deprecate automatic cutout bounds

* pre-commit formatting

* code: Remove cutout dir configuration option

* code: Fix option for archived mobility_profiles

* code: Cleanup unused stub

* config: Remove deprecated config option from configfiles

* code: Remove prompt for API key

* LUISA land cover database - data versioning

* JRC-IDEES dataset versioning implementation

* Remove obsolete script for retrieving jrc idees data

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Scigrid gas network - versioning implementation

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* doc: scigrid dataset updates to versions.csv

* Synthetic electricity demand versioning

* Copernicus Global land service versioning

* changing not-supported to might-work flags for some datasets

* Ship raster - versioning implementation

* EEZ dataset versioning implementation

* NUTS3 population dataset versioning

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* GDP per capita dataset versioning

* WorldPop population count dataset versioning

* EEA ghg emissions data versioning

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* GEBCO versioning

* Attributed Ports data versioning

* Updating URLs

* CORINE dataset versioning

* Emobility dataset versioning

* Common function to unzip folders

* H2 salt caverns data versioning

* Update primary link for GHG emissions

* Primary link update for population count dataset

* Lau regions data versioning

* Data versioning for Aquifer data BGR

* Updating archive link for lau regions

* Retrieving corine dataset from primary datasource

* OSM boundaries primary data versioning

* OSM boundaries archive link update

* rename function

* code: Fix enspreso_biomass dataset name

* code: Improve WDPA dataset handling

* code: Simplify zip file handling

* code: Simplify eurostat balances access

* code: Centrally configure http storage provider

* code: Fix typo

* code: Enable config of prepare_kwargs

* Bug fix - remove duplicate input entries for rule build_shapes

* Removing check for retrieve flag

* Moving corine Apikey from file to config parameter

* Polishing the code

* Code polishing for aquifer data set

* code: Remove unnecessary code

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* code: simplify worldbank retrieval rule

* change indexing method for rules

* Remove code block for retrieving databundle

* modifying the way to unzip folders

* Code modifications

* doc updates

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Script license

* License correction

* Doc formatting

* code: Cleanup GEBCO retrieval

* code: Always copy storage(...) retrieved data, manage keep_local centrally

* code: remove params from retrieve_jrc_idees

* code: remove params from retrieve_gas_infrastructure

* code: remove params for retrieve_eez

* code: Remove params for retrieve_osm_boundaries

* code: Deprecate enable::retrieve, enable::retrieve_databundle, enable::retrieve_cost_data

* code: addendum to centrally managing `keep_local` setting for storage(...)

* code: formatting

* doc: Update configtables for data:: config options

* code: Cleanup old comments

* doc: remove duplicate dataset

* doc: Clarify `url` entry in data/versions.csv

* Code cleanup

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Adding comments

* Versioning for dh_areas

* Data versioning for geothermal heat utilisation potentials

* Update links for archive versions of dh_areas and geothermal heat utilisation potentials

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* doc: Configuration entries for dh_areas and geothermal_heat_utilisation_potentials

* code: Remove obsolete databundle retrieval

* doc: Update retrieve.rst

* doc: Remove outdated information and sites

* doc: Update PR template with hint to instruction on data versioning

* code: Fix config["enable"]["retrieve"] artifacts

* code: Update paths for accessing retrieved gas network

* code: Remove obsolete script

* code: Remove validate_checksum as it keeps failing from too many requests

* code: Fix links for web archive versions

* doc: Update formatting

* doc: try different table formatting

* code: Increase retries on retrieve

* code: Fix in biomass potentials

* code: Modify code to reference variable instead of index

* code: Bug fix

* code: correction to PyPSA set_option

* remove conda in snakemake rule

* code: Set max_requests_per_second

* code: Adding versioning for JRC Ardeco dataset

* code: JRC Ardeco URL updates

* code: plot balance map bug fix

* Update pypsa version to 0.33.2

* code: Cleanup ARDECO retrieval

* code: Fix ARDECO retrieval rules

* code: Adjust http storage limits to avoid Zenodo rate limit

* debug: Extract information about Zenodo cert

* Switch to primary sources for BFS data to avoid web archive retry limits

* code: See if we can extract the Zenodo rate limit

* code: Only request head

* code: Align retrieve_corine with rest of code

* code: Remove superflous parts

* code: Add modified snakemake storage plugin

* code: Fix typo

* wip

* WIP stuff

* code: Bugfix workflow

* nuclear approach to rate limiting errors

* code: Fix EEZ retrieval from archive

* feat: Add snakemake storage plugin for downloading zenodo files

with ratelimit backoff, concurrent downloading and progress bars.

* Update retrieve.smk to rely on zenodo storage plugin

* Make pre-commit happy

* fix url substring sanitization

* Use tqdm-loggable to avoid flooding github logs

* Add retries option to zenodo storage provider

* fix: merge conflict to accomodate new rule build_dh_areas

* fix lau_regions zenodo url

* Re-add zenodo checksums (retrieved from api) and pull retrying into storage plugin

* Replace aiohttp with httpx

better connection pooling and timeout handling.

* Release the stream a tiny bit earlier

* fix: do cache data_versions

for lru_cache to work the method needs to be at the toplevel.

* Allow deactivating the cache and skipping all remote checks

Use these settings for ci, since we already cache the resulting data folder.

* Derive from settings from SettingsBase and clean up some docs

* Revert "debug: Extract information about Zenodo cert"

This reverts commit c1edb28.

* fix: address unfounded fears

* improve docstrings

* feat: improve checksumming by allowing any digest

* Update README.md

* Expand user in `cache` directory

* fix: missed fixes

* cleanup

* refactor: move storage plugin to own repo

* refactor: rename `version_latest` to `version`

* feat: allow passing corine secret via envvar

* refactor: move zenodo cli to dedicated `utils` dir

* Skip storage plugin cache in CI

* feat: extract aquifer_data onaccess

saves space.

* fix: use vsizip to transparently decompress nuts3 2021

* fix: save synthetic electricity demand as csv.gz

* fix: clean conda caches after update

* Revert "fix: save synthetic electricity demand as csv.gz"

This reverts commit 0ebc02c.

* Revert "fix: use vsizip to transparently decompress nuts3 2021"

This reverts commit 9e6cd7f.

* Revert "feat: extract aquifer_data onaccess"

This reverts commit 6de88a2.

* remove tenacity and old checksum/ internet helpers

* Adjust file paths to work on Windows

Minor changes:
- set paths to always use posix format

* Fix file paths on windows nicely (#71)

* fix: use string paths with / even on windows

* feat: Add pixi toml and lock

* fix: fix retrieve_worldbank_urban_population

* Update rules/build_sector.smk

* Revert "feat: Add pixi toml and lock"

This reverts commit 9c9bf97.

* code: Remove quotation marks in config

* Update comment

* Update storage plugin config

* code: Update pixi dependencies

* code: Fix wildcard

* Update rules/build_electricity.smk

Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>

* Update doc/release_notes.rst

Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>

* update `pixi.lock`

* remove snakemake-storage-plugin-cached-http install from pypi

* Revert "update `pixi.lock`"

This reverts commit 6cecd76.

* revert updated lock file

* fix pre-commit

---------

Co-authored-by: Sermisha <sermisha.narayana@openenergytransition.org>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sermisha <156903227+SermishaNarayana@users.noreply.github.com>
Co-authored-by: Lukas Trippe <lkstrp@pm.me>
Co-authored-by: Jonas Hoersch <jonas.hoersch@openenergytransition.org>
Co-authored-by: Jonas Hörsch <coroa@posteo.de>
Co-authored-by: Eric Nitschke <eric.nitschke@tuhh.de>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>

* Fix BFS Swiss transport parsing (PyPSA#1934)

* Fix BFS Swiss transport parsing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add release notes

---------

Co-authored-by: Danele Lerede <danelelerede@Martins-MacBook-Air.local>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Adding Existing Biomass Heating in Urban Decentral and Rural (PyPSA#1872)

* added existing biomass decentral and rural heating

* added release notes

* fixed typo

* fixed typo in definitions

* fixed pre-commit

* Fix build_energy_totals (Swiss transport) (PyPSA#1936)

* Fix 2.0 build_energy_totals for CH.

* Updated release notes.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Set mock_snakemake to rule defaults.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fix bug in marginal and capital cost calculation for tech biomass-to-methanol (CC) (PyPSA#1941)

* chore(deps): bump peter-evans/create-pull-request (PyPSA#1942)

Bumps the github-actions group with 1 update: [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request).


Updates `peter-evans/create-pull-request` from 7 to 8
- [Release notes](https://github.com/peter-evans/create-pull-request/releases)
- [Commits](peter-evans/create-pull-request@v7...v8)

---
updated-dependencies:
- dependency-name: peter-evans/create-pull-request
  dependency-version: '8'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* ci: run model runs only with src changes (PyPSA#1932)

* ci: run model runs only with src changes

* add data

* code: Remove unused inputs from snakefile (PyPSA#1909)

* Update README.md

Updated warning and note sections for clarity and consistency.

* update pypsa-eur logo (PyPSA#1946)

* chore(deps): bump actions/cache from 4 to 5 in the github-actions group (PyPSA#1945)

Bumps the github-actions group with 1 update: [actions/cache](https://github.com/actions/cache).


Updates `actions/cache` from 4 to 5
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](actions/cache@v4...v5)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump the github-actions group with 2 updates (PyPSA#1947)

Bumps the github-actions group with 2 updates: [actions/upload-artifact](https://github.com/actions/upload-artifact) and [actions/download-artifact](https://github.com/actions/download-artifact).


Updates `actions/upload-artifact` from 5 to 6
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@v5...v6)

Updates `actions/download-artifact` from 6 to 7
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](actions/download-artifact@v6...v7)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: actions/download-artifact
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Make model instance conveniently inspectable (separate problem prep and solving) (PyPSA#1902)

* refac: separate problem prep and solving

* feat: add configs for rolling horizon and iterative cases of small BE model instance

* move new rh and iterative configs to examples/

* docs: patch release notes

* fix: enable brownfield + rh mode

* refac: implement all feedback

* remove RH configs

---------

Co-authored-by: Lukas Trippe <lkstrp@pm.me>

* Fix OSM network building from overpass (formerly osm-raw) (PyPSA#1940)

* Fixes for network building from overpass (formerly osm-raw).

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated config.default and toplevel.csv configtable.

* Cleaned up getters in retrieve_osm_data.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated release notes.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>

* Update locked environment files for all platforms (PyPSA#1948)

Co-authored-by: lkstrp <62255395+lkstrp@users.noreply.github.com>

* follow PyPSA to name 'PyPSA-Eur' contributors as authors/copyright (PyPSA#1950)

* Feature: Interactive balance maps (PyPSA#1935)

* Renamed original static plotting to '_static' and changed all map outputs to 'RESULTS + maps/static/...'. Maps in resources are not touched.

* Fix: Updated config_providers to read updated config key 'balance_map_static.'

* Interactive map implementation and static map plotting refactoring.

* removed H2 vmin and vmax.

* Dropped enable toggle.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* removed test config.

* Restore build_energy_totals from master

* Reverted to old plotting keys for balance_maps and changed balance_map_static back to balance_map

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Reordered plotting.default.yaml and added configtables for interactive maps.

* Added release notes.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>

* fix: compatibility of build_gas_* rules with pyogrio >=0.12.0 (PyPSA#1955)

* fix: compatibility of build_gas_* rules with pyogrio >=0.12.0

pyogrio >=0.12.0 returns JSON fields directly as dicts/lists, so that loading them again
as json fails.

refer to https://github.com/geopandas/pyogrio/releases/tag/v0.12.0 .

* Update release_notes.rst

* [pre-commit.ci] pre-commit autoupdate (PyPSA#1954)

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.13.3 → v0.14.10](astral-sh/ruff-pre-commit@v0.13.3...v0.14.10)
- [github.com/fsfe/reuse-tool: v6.0.0 → v6.2.0](fsfe/reuse-tool@v6.0.0...v6.2.0)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: types

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lkstrp <lkstrp@pm.me>

* doc: fix PR template hyperlink (PyPSA#1957)

* fix: add pydeck lower version bound that was missing from pypsa <1.0.6 (PyPSA#1959)

* chore: renormalize line endings to enforce .gitattributes

* chore: renormalize line endings

* fix: fix compatibility with numpy >=2 (PyPSA#1958)

* fix: Version controlled data layer (PyPSA#1963)

* code: Make path relative to current snakefile

* doc: Add release notes on data layer

* feat: define data layer for tyndp specific data

* refactor: use rules attributes to define inputs

* refactor: fix merge conflicts in REUSE.toml

* refactor: make tyndp rules conditional

* doc: integrate doc/data-retrieval.rst information in doc/tyndp-2024-bundle.rst

* refactor: deprecate retrieve_additional_tyndp_data

* chore: fix formatting and improve consistency in data layer doc

* doc: add release note

* refactor: transform pecd release workflow to adapt to data layer

* fix: use input function to define clean_pecd_data input as branch evaluate both branches as dag evaluation time

* chore: adjust reuse.toml to reflect open-tyndp contributors work

* doc: update docstring of dataset_version

* feat: address edge case when no pre-built exists in get_pecd_prebuilt_version

* doc: fix typo in versions.csv

* chore: update lockfile

* [github-actions.ci] Update locked envs

* fix: remove unused carriers from the interactive map configuration

* fix: fix inputs for clean_projects

* fix: adjust dependencies for plotting rules in integration tests

* Apply suggestions from code review

Co-authored-by: Daniel Rüdt <117752024+daniel-rdt@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* doc: add tyndp_pecd entry in doc/data_inventory.csv

* chore: update lockfile

* [github-actions.ci] Update locked envs

* refactor: reduce diff with upstream in common.smk

* doc: improve pre_built definition

* refactor: add option increment_minor in prepare_pecd_release

* Update rules/sb.smk

Co-authored-by: Daniel Rüdt <117752024+daniel-rdt@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: add seawater nc to archive and versions.csv

Cherry-picked from PyPSA@34c0c84

* ci: fix ci cache update

Cherry-picked from PyPSA@17cf106

* fix: fix source typo for seawater

* fix: seawater url

* fix: retrieve eez data from own archive in test (PyPSA#1992)

(cherry picked from commit c49f04c)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Johannes HAMPP <42553970+euronion@users.noreply.github.com>
Co-authored-by: Sermisha <sermisha.narayana@openenergytransition.org>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sermisha <156903227+SermishaNarayana@users.noreply.github.com>
Co-authored-by: Lukas Trippe <lkstrp@pm.me>
Co-authored-by: Jonas Hoersch <jonas.hoersch@openenergytransition.org>
Co-authored-by: Jonas Hörsch <coroa@posteo.de>
Co-authored-by: Eric Nitschke <eric.nitschke@tuhh.de>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
Co-authored-by: Daniele Lerede <daniele.lerede@openenergytransition.org>
Co-authored-by: Danele Lerede <danelelerede@Martins-MacBook-Air.local>
Co-authored-by: Lukas Franken <lukas.b.franken@gmail.com>
Co-authored-by: Bobby Xiong <36541459+bobbyxng@users.noreply.github.com>
Co-authored-by: Philipp Glaum <95913147+p-glaum@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Iegor Riepin <iegor.riepin@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: lkstrp <62255395+lkstrp@users.noreply.github.com>
Co-authored-by: tgilon <22521851+tgilon@users.noreply.github.com>
Co-authored-by: Daniel Rüdt <117752024+daniel-rdt@users.noreply.github.com>
Co-authored-by: daniel-rdt <daniel.ruedt@openenergytransition.org>